-
Notifications
You must be signed in to change notification settings - Fork 259
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable CustomDbtNormalization on k8s #238
Conversation
The purpose of this modifications is to "re-clone" the repo inside the pod if the folder doesn't exists. It reuses the structure of the NormalizationRunner. At this points it uses a custom script to create the profile of bigquery destination. The better form is to reuse the transform-config.py script to create the dbt profile.
@davinchia Can this be used? |
How can we benefit from this update? We would love to run dbt transformation on k8s. |
Small bump on this, would love to have this merged ! |
Is it true? |
It's for internal usage, dbt invocation and the end of pipeline should still work |
|
Your branch is not currently up-to-date with |
This feature was deprecated. |
Your branch is not currently up-to-date with |
Solves the issue 5091 - now, only for bigquery destination.
The purpose of this modifications is to "re-clone" the repo inside the pod if the folder doesn't exists. It reuses the structure of the NormalizationRunner.
At this points it uses a custom script to create the profile of bigquery destination. The better form is to reuse the transform-config script to create the dbt profile (pls someone should help me on that).
What
Describe what the change is solving
It helps to add screenshots if it affects the frontend.
Enable CustomDbtNormalization on k8s
How
Describe the solution
Added the NormalizationRunner steps to DbtNormalizationRunner and enforced the creation of git_repo inside the normalization pod.
Recommended reading order
x.java
y.java
Can this PR be safely reverted / rolled back?
If you know that your PR is backwards-compatible and can be simply reverted or rolled back, check the YES box.
Otherwise if your PR has a breaking change, like a database migration for example, check the NO box.
If unsure, leave it blank.
🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.